MPI/IO on DAFS over VIA: Implementation and Performance Evaluation

In this paper, we describe an implementation of MPI-IO on top of the Direct Access File System (DAFS) standard. The implementation is realized by porting ROMIO on top of DAFS. We identify one of the main mismatches between MPI-IO and DAFS is memory management. Three different design alternatives for memory management are proposed, implemented, and evaluated. We find that memory management in the ADIO layer performs better in situations where the DAFS Provider uses Direct data transfer to handle I/O requests. For the other case of Inline data transfer, it may hurt performance. We propose that the DAFS Provider can expose such implementation information for applications to take full advantage of Inline and Direct data transfers and memory management. Comparative analysis of MPI-IO performance over DAFS, network file system (NFS) and local file system (LFS) shows that MPI-IO on DAFS over VIA on cLAN performs 1.6-5.6 times better than on NFS over UDP/IP on cLAN. The performance of MPI-IO on DAFS is found to be comparable to the performance on local file system. Additional experiments show that MPI-IO nonblocking I/O primitives implemented by DAFS nonblocking operations can completely overlap I/O and computation. These results show that MPI-IO on DAFS can take full advantage of DAFS features to achieve high performance I/O over VI Architectures.